Note

This page corresponds to a Jupyter notebook you can try out yourself.
(The original version is here.)

https://mybinder.org/badge_logo.svg

Query Tutorial

[1]:
# The plotting examples below require holoviews, hvplot, and bokeh:
# conda install -c conda-forge bokeh holoviews hvplot
import numpy as np
import pandas as pd

import bokeh
import hvplot.pandas
import holoviews as hv

import bokeh.palettes
from bokeh.plotting import figure, show, output_notebook
output_notebook()
Loading BokehJS ...

Create a Client

Before you begin, you must create a Client object. It will be stored globally and used for all communication with the neuprint server.

Initialize it with your personal authentication token. See the Quickstart guide for details.

[2]:
from neuprint import Client

TOKEN = "" # <--- Paste your token here
           # (or define NEUPRINT_APPLICATION CREDENTIALS in your environment)

c = Client('neuprint.janelia.org', 'hemibrain:v1.0.1', TOKEN)

ROIs

In neuprint, each neuron is annotated with the list of regions (ROIs) it intersects, along with the synapse counts in each.

The ROIs comprise a hierarchy, with smaller ROIs nested within larger ROIs. Furthermore, primary ROIs are guaranteed not to overlap, and they roughly tile the entire brain (with some gaps).

For a quick overview of the ROI hierarchy, use fetch_roi_hierarchy().

[3]:
from neuprint import fetch_roi_hierarchy

# Show the ROI hierarchy, with primary ROIs marked with '*'
print(fetch_roi_hierarchy(False, mark_primary=True, format='text'))
hemibrain
 +-- AL(L)*
 +-- AL(R)*
 +-- AOT(R)
 +-- CX
 |   +-- AB(L)*
 |   +-- AB(R)*
 |   +-- EB*
 |   +-- FB*
 |   +-- NO*
 |   +-- PB*
 +-- GC
 +-- GF(R)
 +-- GNG*
 +-- INP
 |   +-- ATL(L)*
 |   +-- ATL(R)*
 |   +-- CRE(L)*
 |   +-- CRE(R)*
 |   +-- IB*
 |   +-- ICL(L)*
 |   +-- ICL(R)*
 |   +-- SCL(L)*
 |   +-- SCL(R)*
 +-- LH(R)*
 +-- LX(L)
 |   +-- BU(L)*
 |   +-- LAL(L)*
 +-- LX(R)
 |   +-- BU(R)*
 |   +-- LAL(R)*
 +-- MB(+ACA)(R)
 |   +-- MB(R)
 |   |   +-- CA(R)*
 |   |   +-- PED(R)*
 |   |   +-- a'L(R)*
 |   |   +-- aL(R)*
 |   |   +-- b'L(R)*
 |   |   +-- bL(R)*
 |   |   +-- gL(R)*
 |   +-- dACA(R)
 |   +-- lACA(R)
 |   +-- vACA(R)
 +-- MB(L)
 |   +-- CA(L)*
 |   +-- a'L(L)*
 |   +-- aL(L)*
 |   +-- b'L(L)*
 |   +-- bL(L)*
 |   +-- gL(L)*
 +-- OL(R)
 |   +-- AME(R)*
 |   +-- LO(R)*
 |   +-- LOP(R)*
 |   +-- ME(R)*
 +-- PENP
 |   +-- CAN(R)*
 |   +-- FLA(R)*
 |   +-- PRW*
 |   +-- SAD*
 +-- POC
 +-- SNP(L)
 |   +-- SIP(L)*
 |   +-- SMP(L)*
 +-- SNP(R)
 |   +-- SIP(R)*
 |   +-- SLP(R)*
 |   +-- SMP(R)*
 +-- VLNP(R)
 |   +-- AOTU(R)*
 |   +-- AVLP(R)*
 |   +-- PLP(R)*
 |   +-- PVLP(R)*
 |   +-- WED(R)*
 +-- VMNP
 |   +-- EPA(L)*
 |   +-- EPA(R)*
 |   +-- GOR(L)*
 |   +-- GOR(R)*
 |   +-- IPS(R)*
 |   +-- SPS(L)*
 |   +-- SPS(R)*
 |   +-- VES(L)*
 |   +-- VES(R)*
 +-- mALT(L)
 +-- mALT(R)

Neuron Search Criteria

Specify neurons of interest by bodyId, type/instance, or via a NeuronCriteria object. With NeuronCriteria, you can specify multiple search constraints, including the ROIs in which matched neurons must contain synapses.

[4]:
from neuprint import NeuronCriteria as NC

# Example: Select a specific body
criteria = 387023620
criteria = NC(bodyId=387023620)

# Example: Select several bodies
criteria = [387023620, 387364605, 416642425]
criteria = NC(bodyId=[387023620, 387364605, 416642425])

# Example: Select bodies by exact type
criteria = 'PEN_b(PEN2)'
criteria = NC(type='PENPEN_b(PEN2)')

# Example: Select bodies by exact instance
criteria = 'PEN(PB06)_b_L4'
criteria = NC(type='PEN(PB06)_b_L4')

# Example: Select bodies by type name pattern
criteria = NC(type='PEN.*', regex=True)

# Example: Select bodies by region (input or output)
criteria = NC(rois=['PB', 'EB'])

# Example: Select traced neurons which intersect the PB ROI with at least 100 inputs (PSDs).
criteria = NC(inputRois=['PB'], min_roi_inputs=100, status='Traced', cropped=False)

Fetch neuron properties

Neuron properties and per-ROI synapse distributions can be obtained with fetch_neurons(). Two dataframes are returned: one for neuron properties, and one for the counts of synapses in each ROI.

[5]:
from neuprint import fetch_neurons
neuron_df, roi_counts_df = fetch_neurons(criteria)

The total count of pre-synaptic and post-synaptic points within each neuron are given in the pre and post columns:

[6]:
neuron_df[['bodyId', 'instance', 'type', 'pre', 'post', 'status', 'cropped', 'size']]
[6]:
bodyId instance type pre post status cropped size
0 387023620 PEN(PB06)_b_L4 PEN_b(PEN2) 343 1714 Traced False 685261005
1 387364605 EPG(PB08)_L3 EPG 645 4333 Traced False 1459837829
2 416642425 EPG(PB08)_R4 EPG 470 3828 Traced False 1165123520
3 417998204 PFNv(PB05)_R4 PFNv 229 990 Traced False 473864774
4 419060083 PFGs(PB10)_L2 PFGs 556 2395 Traced False 1103881970
... ... ... ... ... ... ... ... ...
476 5813080979 PEN(PB06)_a_L5 PEN_a(PEN1) 544 1595 Traced False 924457147
477 5813081494 PFR_b(PB11b)_R1 PFR_b 118 1891 Traced False 588428634
478 5813081686 PFNa(PB03)_L8 PFNa 121 510 Traced False 283961924
479 5813087532 IbSpsP(PB17)_R9 IbSpsP 108 388 Traced False 257354274
480 5813128308 PFNp(PB01)_a_L7 PFNp_a 71 310 Traced False 187208043

481 rows × 8 columns

The per-ROI synapse counts are returned in the second DataFrame.

Note: Since ROIs overlap (see hierarchy above), the sum of the per-ROI counts for each body will be more than the pre and post columns above.
[7]:
roi_counts_df
[7]:
bodyId roi pre post
0 387023620 CX 341 1704
1 387023620 PB 1 777
2 387023620 PB(L5) 0 33
3 387023620 PB(L4) 1 723
4 387023620 INP 0 5
... ... ... ... ...
7289 5813128308 FB 42 33
7290 5813128308 FBl1 42 33
7291 5813128308 NO 27 107
7292 5813128308 NO(R) 27 107
7293 5813128308 NO3(R) 27 107

7294 rows × 4 columns

Fetch connections

Find synaptic connection strengths between one set of neurons and another using fetch_adjacencies().

The “source” and/or “target” neurons are selected using NeuronCriteria. Additional parameters allow you to filter by connection strength or ROI. Two DataFrames are returned, for neuron properties and per-ROI connection strengths.

[8]:
from neuprint import fetch_adjacencies, NeuronCriteria as NC

# Example: Fetch all downstream connections FROM a set of neurons
neuron_df, conn_df = fetch_adjacencies([387023620, 387364605, 416642425], None)

# Example: Fetch all upstream connections TO a set of neurons
neuron_df, conn_df = fetch_adjacencies(None, [387023620, 387364605, 416642425])

# Example: Fetch all direct connections between a set of upstream neurons and downstream neurons
neuron_df, conn_df = fetch_adjacencies(NC(type='Delta.*', regex=True), NC(type='PEN.*', regex=True))
[9]:
conn_df.sort_values('weight', ascending=False)
[9]:
bodyId_pre bodyId_post roi weight
179 910783961 5813070465 PB 184
57 880880259 849421763 PB 141
316 911565419 5813070465 PB 141
92 910442723 849421763 PB 139
216 911129204 724280817 PB 127
... ... ... ... ...
351 911578496 724280817 PB 1
349 911578496 664645558 PB 1
348 911578496 634608104 ATL(L) 1
346 911578496 509410587 PB 1
660 5813061383 5813070465 PB 1

657 rows × 4 columns

Merge arbitrary neuron properties onto the connection table with merge_neuron_properties()

[10]:
from neuprint import merge_neuron_properties

conn_df = merge_neuron_properties(neuron_df, conn_df, ['type', 'instance'])
conn_df
[10]:
bodyId_pre bodyId_post roi weight type_pre instance_pre type_post instance_post
0 734581598 387023620 PB 1 Delta7_a Delta7(PB15)_a_L PEN_b(PEN2) PEN(PB06)_b_L4
1 734581598 508793049 PB 1 Delta7_a Delta7(PB15)_a_L PEN_a(PEN1) PEN(PB06)_a_L7
2 734581598 509410587 PB 3 Delta7_a Delta7(PB15)_a_L PEN_a(PEN1) PEN(PB06)_a_R4
3 734581598 569775058 PB 17 Delta7_a Delta7(PB15)_a_L PEN_b(PEN2) PEN(PB06)_b_R4
4 734581598 570461892 PB 21 Delta7_a Delta7(PB15)_a_L PEN_a(PEN1) PEN(PB06)_a_R3
... ... ... ... ... ... ... ... ...
652 5813061383 1508334312 PB 1 Delta7_b Delta7(PB15)_b_R PEN_a(PEN1) PEN(PB06)_a_L3
653 5813061383 1600083889 PB 37 Delta7_b Delta7(PB15)_b_R PEN_b(PEN2) PEN(PB06)_b_L9
654 5813061383 1601771527 PB 4 Delta7_b Delta7(PB15)_b_R PEN_b(PEN2) PEN(PB06)_b_R9
655 5813061383 5813047157 PB 12 Delta7_b Delta7(PB15)_b_R PEN_b(PEN2) PEN(PB06)_b_L2
656 5813061383 5813070465 PB 1 Delta7_b Delta7(PB15)_b_R PEN_b(PEN2) PEN(PB06)_b_L3

657 rows × 8 columns

Connection Matrix

You can convert a connection table into a connectivity matrix via connection_table_to_matrix().

[11]:
from neuprint.utils import connection_table_to_matrix

matrix = connection_table_to_matrix(conn_df, 'bodyId', sort_by='type')
matrix.iloc[:10, :10]
[11]:
bodyId_post 634608104 1508334312 664645558 509410587 5813056953 508793049 1125964630 910447075 849482511 847336755
bodyId_pre
850182049 0 0 0 0 0 0 0 0 0 0
1067331159 0 0 0 0 0 0 0 0 0 0
1129396529 0 0 0 0 0 0 0 0 0 0
755950242 0 0 0 0 0 0 0 0 0 0
1225601395 0 0 0 1 1 1 0 0 0 0
1009043211 0 0 0 0 0 0 0 0 0 0
946300031 0 0 0 0 0 0 0 0 0 0
911906936 20 13 0 0 0 0 0 0 0 15
911901802 0 4 0 0 1 0 0 0 33 17
911911699 37 16 0 0 0 0 0 0 1 0
[12]:
matrix.hvplot.heatmap(height=600, width=700).opts(xrotation=60)
Data type cannot be displayed:
[12]:
[13]:
matrix = connection_table_to_matrix(conn_df, ('bodyId_pre', 'type_post'))
matrix.hvplot.heatmap(height=600, width=400)
Data type cannot be displayed:
[13]:

Synapses

Fetch synapses for a set of bodies using NeuronCriteria, and optionally apply additional filtering with SynapseCriteria.

[14]:
from neuprint import fetch_synapses, NeuronCriteria as NC, SynapseCriteria as SC

neuron_criteria = NC(status='Traced', type='FB4Y', cropped=False, inputRois=['EB'], min_roi_inputs=100, min_pre=400)
eb_tbar_criteria = SC(rois='EB', type='pre', primary_only=True)
eb_tbars = fetch_synapses(neuron_criteria, eb_tbar_criteria)
[15]:
# Plot the synapse positions in a 2D projection
p = figure()
p.scatter(eb_tbars['x'], eb_tbars['z'])
p.y_range.flipped = True
show(p)

Synapse Connections

Here, we fetch all synapse-synapse connections from a set of neurons. Provide a NeuronCriteria for the source or target neurons (or both) to filter the neurons of interest, and optionally filter the synapses themselves via SynapseCriteria.

[16]:
from neuprint import fetch_synapse_connections
eb_syn_criteria = SC(rois='EB', primary_only=True)
eb_conns = fetch_synapse_connections(neuron_criteria, None, eb_syn_criteria)
eb_conns.head()
[16]:
bodyId_pre bodyId_post roi_pre roi_post x_pre y_pre z_pre x_post y_post z_post confidence_pre confidence_post
0 5813027016 978733459 EB EB 26199 23565 18002 26210 23553 18003 0.977 0.947756
1 5813027016 978733459 EB EB 25559 25038 23046 25554 25038 23056 0.928 0.920682
2 5813027016 978733459 EB EB 23472 24677 21980 23464 24674 21970 0.683 0.692000
3 5813027016 978733459 EB EB 28490 25331 22543 28519 25332 22545 0.956 0.752377
4 5813027016 978733459 EB EB 29141 24584 19235 29129 24559 19240 0.953 0.334432

Let’s determine the post-synaptic neuron types, and plot the synapses for the top 5 types.

[17]:
# Retrieve the types of the post-synaptic neurons
post_neurons, _ = fetch_neurons(eb_conns['bodyId_post'].unique())
eb_conns = merge_neuron_properties(post_neurons, eb_conns, 'type')

top5_counts = eb_conns['type_post'].value_counts().head(5)
top5_counts
[17]:
EPG            1480
EQ5             571
PEN_a(PEN1)     374
ExR7            264
ExR4            230
Name: type_post, dtype: int64
[18]:
colormap = dict(zip(top5_counts.index, bokeh.palettes.Category10[5]))
points = eb_conns.query('type_post in @top5_counts.index').copy()
points['color'] = points['type_post'].map(colormap)

p = figure()
p.scatter(points['x_post'], points['z_post'], color=points['color'])
p.y_range.flipped = True
show(p)

Skeletons

Download skeletons with Client.fetch_skeleton().

[19]:
# Download some skeletons as DataFrames and attach columns for bodyId and color
skeletons = []
for i, bodyId in enumerate(eb_conns['bodyId_pre'].unique()):
    s = c.fetch_skeleton(bodyId, format='pandas')
    s['bodyId'] = bodyId
    s['color'] = bokeh.palettes.Accent[5][i]
    skeletons.append(s)

# Combine into one big table for convenient processing
skeletons = pd.concat(skeletons, ignore_index=True)
skeletons.head()
[19]:
rowId x y z radius link bodyId color
0 1 21626.0 34402.0 12790.0 10.0000 -1 5813027016 #7fc97f
1 2 21626.0 34402.0 12810.0 10.0000 1 5813027016 #7fc97f
2 3 21626.0 34382.0 12830.0 10.0000 2 5813027016 #7fc97f
3 4 21626.0 34362.0 12850.0 18.2843 3 5813027016 #7fc97f
4 5 21606.0 34342.0 12870.0 18.2843 4 5813027016 #7fc97f
[20]:
# Join parent/child nodes for plotting as line segments below.
segments = skeletons.merge(skeletons, 'inner',
                           left_on=['bodyId', 'rowId'],
                           right_on=['bodyId', 'link'],
                           suffixes=['_child', '_parent'])
[21]:
p = figure()
p.y_range.flipped = True

# Plot skeleton segments (in 2D)
p.segment(x0='x_child', x1='x_parent',
          y0='z_child', y1='z_parent',
          color='color_child',
          source=segments)

# Also plot the synapses from the above example
p.scatter(points['x_post'], points['z_post'], color=points['color'])

show(p)

Cypher Logging

Tip: Inspect all cypher queries by enabling debug logging.

[22]:
from neuprint.client import setup_debug_logging
setup_debug_logging()

synapses = fetch_synapses(5813027016, SC(rois='EB', type='post'))
[2020-06-30 05:18:46,776] DEBUG Performing cypher query against dataset 'hemibrain:v1.0.1':


    MATCH (n:Neuron)
    WHERE
      // -- Basic conditions for segment 'n' --
      n.bodyId = 5813027016
    RETURN n.bodyId as bodyId

[2020-06-30 05:18:46,811] DEBUG Performing cypher query against dataset 'hemibrain:v1.0.1':

    MATCH (n:Neuron)
    WHERE
      // -- Basic conditions for segment 'n' --
      n.bodyId = 5813027016
      AND (n.`EB`)

    MATCH (n)-[:Contains]->(ss:SynapseSet),
          (ss)-[:Contains]->(s:Synapse)

    // -- Filter synapse 's' --
    WITH n, s
    WHERE (s.`EB`) AND (s.type = 'post')

    // De-duplicate 's' because 'pre' synapses can appear in more than one SynapseSet
    WITH DISTINCT n, s

    RETURN n.bodyId as bodyId,
           s.type as type,
           s.confidence as confidence,
           s.location.x as x,
           s.location.y as y,
           s.location.z as z,
           apoc.map.removeKeys(s, ['location', 'confidence', 'type']) as syn_info